Efficient algorithms for exact hierarchical clustering of huge datasets: Tackling the entire protein space

Motivation: UPGMA (average-linkage clustering) is probably the most popular algorithm for hierarchical data clustering, especially in computational biology. UPGMA however, is a complete-linkage method, in the sense that all edges between data points are needed in memory. Due to this prohibitive memory requirement UPGMA is not scalable for very large datasets. Results: We present novel memory-constrained UPGMA (MCUPGMA) algorithms. Given a constrained memory size, our algorithm guarantees the exact same UPGMA clustering solution, without explicitly holding all edges in memory. Our algorithms are general, and applicable to any dataset. We present a theoretical characterization of the algorithm efficiency, and hardness for various data. We show the performance of our algorithm , under restricted memory constraints. The presented concepts are applicable to any agglomerative clustering formulation. We apply our algorithm to the entire collection of protein sequences, to automatically build a novel evolutionary tree of all proteins using no prior knowledge. We show that newly created tree captures protein families better than state-of-the-art large scale methods such as CluSTr, ProtoNet4, or single-linkage clustering. The robustness of UPGMA improves significantly on existing methods, especially for multi-domain proteins, and for large or divergent families. Our algorithm is scalable to any feasible increase in sequence databse sizes. Availability: The evolutionary tree of all proteins in the entire UniProt set, together with navigation and classification tools will be made available as part the ProtoNet service. A C++ implementation of the algorithm, suitable for any type or size a data, is available. Contact: [email protected]